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4 The way that the U.S., Department of Iducation's 

Of f ice of Student Financial,' Assistance (OSFA) should conduct its 
^ality assurance* function of monitoring a processing contractor 
described. For the' following quality assurance procedures, 
information is prdvided on thS goat of the. procedure, activities" 
be measured, measures, and the role of quality assurance: production 
control, information receipt^ dat? entry, data' edit i computation, 
document product-ion and mailing, corrections, fiscal control, 4 
software qua).ity assurance, productivity control, telephone service,, 
correspondence, reporting process, and corrective action process. ' 
Alsp included is a sum&ary checklist of qua! ityv assurance, proc^ures 
that monitors can use to assure they have completed all quality 
assurance tasks. Appended are: a summary of the division of 
respossibility between the contractor and OSFA for maintaining the 
qiAality df ^oftwar^; a monitor's checklist for reviewing the software- 
developed by the^processing. contractor; an esample of a signoff form 
to' notify OSFA of proposed changes in the system or system software; 
an example of an ^Stk internal r^eporting form that summarize^ Key 
processing, measures and exceptions; and an example of OSFA interna^ 
reporting summary form (section 1) tailored to Pell Grant processing. 
(SM) • . 
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1^ INTRODUCTION V ^ ^ ^ , * 

■ ■ , • . . , "■ „.., ■ '■ '.* • ■ ■ ■ ■ . ■ ^ 

The processing of student assistance applications is a cpmplex endeavor. It is 
imperative that each step in the processing procedure 'be carried o«t ini an accurate 
and timely mtoner. , To ensui*e such accuracy arid tiitieiiness, the processing 
'procedure must be monitored on an ongoing basis. This function is shared iy the 
coaitractor performing the processing and OSFA. . -The process 6y which thp 
contractor, establishes and rhaiht^lns procedures lor monitoring its own activities is 
5:aUed ({uamy GontraJU The process by whi(dv OSFA i^nonitors tiie quality of contract 
perlw-mar^ce is caUed^QitaUtt askiranc^ \ 



V - 



This report describes^how OSFA should carry out its qualify assurance fjOnctiw 
of monitoring a pfpcessing contractor. It, is a companior\ piece to an earlier vtsiuftie, 
."A f^ratfnework f or a Quality Control System Vendor/Pfocessor Omtracts" 
(Advanced Technology, lnc.» September, 1982), ivhich detailed the critical cjuaiity 
control activities tiiat should be performed by the contractor. Quality control 
procedure* are fii requlremertVin all large p^oceSsii\g t;ontr^ 

. ' \ / r 

The two reports are organized^in a simUar m^er/ ' Tbey bioth break down the 
respective c6nttr^ct<^^^^ c^^ 
categories; 4 ; . >^ 



Production contFoi*' 
Fiscal control ' - 
Software qiklky 

ProdtKttivity Qontrol v 
Reporting process 

.....vQ''r^!?!*il!Y®.!*5^.^9",Pf*f'^ : ,:-:......* .' 

The, first four categories comprise the components of the processing system 
4hpt must be mcaUtored f or asxuracy and efficl^wry.' ^The rq^ortir^ process 6nd 
corrective action process repre^nt functions thf^t interface with ti>e other compo- 
nents of the proce^iftg system. - 



, V Since productiofji control is particularly crucial to the int'fegnty of . the 
processing system, thisfcategory is further separated IntQ siic discrete st^s:. 



Iniormatiorf receipt 
Datajentry 
Data^dif 



I. 



Compute 
Qocument broducjtioh and mailing 
Gorrectiofi proc^sdures. 



Also, productivity coatrol is broken out into: 

f t ■ * ■ ' ■ -."^ ■ 

« Telephone service 
• ^ Corre^cmdence service' 
Irrformi^o*^ f ecelpt 



Data^ 

Documejfit p^oAictioh and maJlihg 
» 

^rect ons receipt. » 



These discrete prccessing steps are also. discussed in_the ^lier quality control 



report. 



Although similfir in categorical content, the companion reports are formatted 
in different mannej-^ The quality control report is a narrative explaining in detail 
the various steps in the quality control process, this quality assurance report 
provides an abstract of Tcey issues in a sumrtiiary was ^osen 

because the; principal purpose of the report is to serve as a reference source fof 
qualltjf assurance m<»tlt<^s. The report ^6uid ^rve as a hamftjook that distir^ly 
highlights the various quality assurance responsfijifities of OSFA. 




.the information ^.in this repm intended to meet the needs of quality 
ass&rance irionitors working oh various pfocessing' contrAct§. To accomfjlish this, 
generic categories ' typical of student assistance ; processing , are used w^enevfer 
possible. For ex^mpie,.^he various quality assurance control, fiscal 

soft^Ware qu§J|ty assuraiite^ produ?:tlvity .control, reporting, corrective 

■ ■ ..... 



action) and proii^B^RVS^teps Cinlofmation receipt, data entry, data edit, compute, 
document pl^oauc^ioh ^8 mailing, corrections) descriijed in this report are 
components Ol fyiost pr^oces^R'contract^ . . ^ ^ 

Not aii ia^^ assurance roles and processing steps will* he relevant to alF 
contraptsi i^0«6ver* -A st^rdent ihiormation telephone servicp which is a part "of 
productivity control, and i prin6ipa^ ^ciliary service- in the Pell Grant processing 
comract, 1^^; h^^^ the GSL 'or Gampus*Based programs. Therefore, 

> mon!fors isfitoiiid'tise #iis handbook as a model or guide and. adapt relevant Sections to 
help therri iuif ill their qua^ 

In iiddition, it i$*passiile ^t sOme proceWin© fw<^ons that are unique to a 
particul^C'i''^ bej addressed in -bieTeport^ the functions ih^t are 'mc|.uded 

4re basei;4^ analysis ofVthe PeU prpc^ssihg contract, ^ s^ it is^ the largest 
student a^i^^lnce j!>r^^ It huist^^^^^^ is ^ 

rriodel fbr iJtsH assurance monitors and in sbme instances ^ould be used 

. \^'-'v:-.;.^X''v'->. , ■■■■ ^ . . ■ :■'<:. ^ . ' . . , • . , • 

M Org^hiz^iion of the Report 

i^oilowitisg ^is intrSuction, Section 2.0"qf the rep<Mt is a narrative explanation 
of the Foie of ^urance. The balance of the report utilizes the summary 

fornfiat.^^ .Seetidn 3^0 describtes specific quality assurance procedures Jor the 
following cat^ories and siijoategories: ' > * 

• Prod^CI^Oft control ' ' ' ' ' ^ 

. Ittfe^ftii^bh receipt ' ' 

' ' • Data entry 




JDocument pfl)duction and^handiing 
■ Corre'ctipnS' ' ■ , .-V-\. 
Fiscal control 
Software quality, assurance 
Productivity control 
Teiephone service 
'Correspondence service . 
Reporting process 
Corrective action process. - 



Section ^.0 presents general, ongoing quality assurance procedures. Section 5.0 is a 
summary checklist tear-Out sheet of quality assurance procedures that monitors cajs 
use to assure tijey have comj^ieted all selected quality assurance tasks. 

.... „.' V.. .:.V;, : ; ■ : . 

The report also ioiGludes* a series of ^ppertdicefs* Appendix A ^mmaf izeS the 
division of responsibility between" the ; contactor and OSFA fOY maihtalfiihg the 
quality of software. Appendix B is' a monitor's checklist for reviewing the. «>ftware 
developed by tiie j^ocessing c^^^ C is an example of a^signotf form 

to notify OSFA of proposed changes in the system or system software. Appendix D 
is an example of an OSFA Intori^ retJorting iotm that summariSEes key processing 
measures and excef>tions. Appenclix E shows how Section i of tfjo^SFA internal 
reporting summary ^orm pan be taaored to p. specie contfSct, in ^is case PeU 
Q^ant processing. * f - ^ 



a.0\ THE ROlE bF QUALrrY te 

■* , / ^■■■v:/^ ; . . ■ ' . 

\T^e student aid delivery system is composed of various components ranging 
from ^Jtamting to account -reconciUation. Perhaps the most complex component is 
the processing function. This function includes the receipt and processing of student 
aid^ applications and the production of an output, such as a Student Aid Report (SAR)' 
in the tjase of the Peli Grant Program. One or more steps in the processing function 
♦is 'tisuaUy automated. The complexity of the function is attributable to Jiae large 
number of steps involved , irt the process^ the mix .of automatesl aiSiarianual 
proccciures,V and the great ^ume of documents to be processed* , fri order to 
maintain maximum efficiency from this complex f^ctionV effective management is 
mandatory- on a contusing basis. - » ' 

Management of the processing function comes from two spurces-si|the 
contractor selected by 6SFA^to run the processing system and OSFA itself^ tfvt 
contractor is responsible for monitoring the everyday ftmctipning of tKe itfBcessing 
operatic^ and ensuring' that^^ ™ 

l3-actd| role is called quaU^ cotitroi. OSFA* is responsible for establishing 
inSdependfent performance standards that must be met. by the contractor and 
ntmiiiorihg whet^^^ sysjiftm performance actually meets these standeurdA.^^ ^^^^^"^^ 
contracting agency role is called <iuallty asstarance* . 

/ A quality control plan serves several purposes. Quality control can help 

» ■ ■ - ■ . . , . . # ' ' ' 

ensures . > ' * 

■ ' ■ - . . . . , . 

\ ' 9 Timely a^d -accurate pcpducts 

• Cost-effective performance by th« contractor , 

• Fiscal integrity of the project . 

" QuaUty service to-st^ents, parents, institutions, afK^^ , 

• Timely and accurate information to OSFA *. 



Credibility of ^ entire stud^t jpsistance delivery system-. , 



.« 5ach of the^ diaracteristics is disaissed in tumj.. Firsti timely, and accurate 
products are impprtaint sln^ the principal role of tiie processii^ function is to 



^projduce an output document. That i§, the processing function accept^ a great deal 
of data, processes data, transf orms the data, and produces an output document. 
The primary output^^y^^ell Grant processing functldn, for,example, is the SAR. 
It is imperative that this document be computed correctly and mailed to the student 
In a timely manner. Other output documents, also requiring timely and acct^rate 
production, may include applicant rosters -and; data tapes for institutions and state 

y^encies- Quality control procedures can monitor the production:4jrpc measure 
the timeliness and accuracy of output, ^ Compare the measires * to established 
standards, and prescribe corrective'act|ons Avhen necessary. ^ 

Second, it is imperative- that production be carried out in a cost-effective 
manner by tiie processor. Givisn the high volume of applications and the depen" 
dencies of processQig procedures on previous isteps,» even small production 

. inefficiencies can snowball i^to major cost problems. Quality control methods can 
provide information on the quantity of oujput and the levels of input (intluding labor, 

^ hours, cpmputer time, afid ddilars) used to generate t^^^ so that cost 

eft^tiveness caivbe ccjntinuaUy mofiltored and corrective' actions instituted if 
measures fall beloW standards*, • . V - 

* , A third roje of quality cohtrotl can be to prdtecit the fiscal Integrity of the 
project by keeping spending within program budget. A larg^ production process is 
prone to cost overryns. Quality control proc^dt^^ <:bi\ monitdt actual costs anef 
compare^these costs to budgeted costs for eachSjost center as Well as 'for iJhs 
contract as a whole« Statistics can be generated for each reporting period and for 
the c<»itfact fwrlod to date. , * 
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• Four^, quality control methods can abo he^m^imlze'th^ quality of service 
to stu dents, parents, institutions, ^d lenders. For example, anCiUary services, such 
as telephone* banks, can be monitored to ensure that questions from u'sefs receive 
-^curate arid helpful "riK^hises from "courtemis- operators*'' Also, Information on an 
institutional roster can be rechecked for accuracy and reported* in, a format that 
meet^ the heetfe of Instl^tibns* • 

' . * ■'/•,' . ' ' ■ ■ / ' , . ~ ■ ' ' ' ■ . 

Fifth, it is important that tinjily accurafe reports and statistics On the 
processif^ ftBiCtion result from the quality control procasdure and are made available 



to OSF A management. These reports and statistics should compare performance to 
standards, provide frequency ^distributions as well as averages since the latter trfeiy 
hide significant trends, and inciude a visual device to fiag"data falling outjsjde of 
toterance levels. AH statistics and reports' should be at level of specificity, kntL 
formatted so that the^ are mdst respdnslve to OSFA's maaagement needs. * 



( 



Finally^ Hivt processing function is only" one of several parte of a students aid 
system. dther.n?ajpr.pam include policy det6rminatk)ni budget formulatioh, 
' awarding of aid, and fiscal reconciliation. The delivery of student assistance ts' 
dependent upon the efficient interaction of j^U these parts. The various systeiln 
functions can run , either se<|uent^y or 5imultaneously# If two functions , are 
sequential* informatio;^ from one part is needed before another part can beg^n 
operation. If the parts run simultaneously* they follow different tracks at the same 
time but may h^ye several c;pmmon points between them where information is 
passed from one function to another, in either case, the system must run smoothly 
to deliver the programs effectively. If a particular deci^ioifis delayed>'for instance, 
the pther functions may not- be able to perform, theif tasks.* If Infori^tibn passed 
from one function to >inc«her/^is Jnaccurate, the error will be c6mpot|nded by. 
subsequent parts of the delivery system. Therefore,, a well-designed quality control 
ipian does not just ensure accurate and tiinr^Iy procesjs^ the 
credibility of the entire delivery system: , , ^ ' 

While quality control procedures are essenti^ to tiie efficient operation of ^ 
processing system,/ they are not sufficient from i management perspective. OSFA 
<annot rely exclusively on the reports and statistl<^ generated by the contractor as' 
proof of processing efficiency* In s«ne cases, If processif^ problems exist, the 
contractor may not- make eVident all problems, w the full extent of the problems, 
known to OSFA* Therefor«|, OSFA must perform its own checks on system 
perforrnance. , 




It is important to note that OSFA does not have the resouross to implemerit a 
quality assurance system that is as tomprehensive as the quality control process 
designed by the contractor! nor is ' it in OSFA's best interests^ to create a. 
comprehensive quality a^surancf system that merely duplicates' the procedures 
performed by the contractor. An effective <^Uty assurance progfaf|j ^ould be 
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complementary, to the quality control function. OSF A 'should -utilize extefis'ivety. the 
reports and statistics produced, by the quality, control process* To be suse^tt^t this 

"information is* accurate, OSF^^ primary quality fTssurance task sHoiild be to validate 

<^ese data.. Va^dation/can be accomplished by:^ 

■.■ : ::„:;■ ■ : -„ :■■• ' 

■ ■ ■■; ■ . ' ' V .' , / ■ .. 

' • Moriitofihg the contractor's quality 'corttrof^profedures / 
/• Conducting independent checks on system performance, ' . 

. "^onitoriiig the contractor^ <?uaUty . cohtft>I procedures entaUk verifym^ the 
processes used to produce the quality coi^troi statistics and reports- Specyically, 
^e monitor should; - , , , ' v 

• Review the contractor's quality control plan including the type of quality 
* , * Control procedures used, the frequency of the^= procedures, and 

" . particular measures of performarice'^ , ^ 

• Note problems ^d/or deficiencies in the* system based on observatii 
the pretducSoFVlJirocess' . ' 

• Compare performance of the system based on observation to th* 
iltformation in the quality control reports . ^ 

• Review the contractor's plan to implement corrective actions in the! 
y . event of system problems 

• Review the. <^ tractor's control reports with an emphasis on 
\ comparii^ nfeasures with standards* timeUh 

responsiveness of the reports^ and the* use of clear visual cues to idehtif y^ 
problem, areas • 

• » Eftetire that tJw contractor is analyzing prodsssing trends so ^^t 

potential problems are idenftftfifd before they become serious problems 

• Ensure that the contractor is not solely' ctMicerned with maintaining the 
* status quo, bqt Is atso looking to make ^ystem^ breaktiiroi^h^ that will 

. increase overall efficiency, ♦ . • " ,' 

These latter tl^ee points require elaboratiwi. First, tbe contractor's reports 
provide a critical source of inf orrnatien to the quality assurahc^ monitor. It is 
Important that the time between ^^erati^^^o^^ cfeta ami si#miJal<m ol report to 
SFA be' minimized to ensure the avaUabllity of timely clata. Edition, to be 
nwst respcffisive to OSF A«s n^s, tl» reports must tailor level of cfetaU to 
different management levels. High-level managers sh«ild only be presented 



sij^^ary data such as information 'on selected key measures and exceptions. Middle 
i^Syw^-^fevel management require increasingly more detail and a greajer numbec of 
pcf^oTTtiaric^ measures! Further,^ the reports should include a clear system alarm 
cap^Uty t<^ flag performance that is outside of established tolerance levels. , 

Second* a contractor or quality assurance monitor should not rely solely on a 
system alarm capability t<i flag problems.^ Trend data (e.g', data on performance ior, 
this rep6rting period, prior reporting perio{J,s, arid the ♦year to dat^) should be 
carefully analyzed to identify potentisfel problej are^ bef ^ 

• ■ ■ ' ■' ■ ■ ' . ■ ' - • 

fhi^d, the quality assurance monitor should not solfely, require the contractor 
to maintain acceptable-minimum operating proce^fcires. Instead,' the ! monitor sJiouid 
(sncourage system simplification- and the dfrvelopment of new system options that 
'can ihcrisa«!^ production efficiency. - ' V 

' - Conducting . independent checte on the processing system's performance is 
OSFA's assurance that the system isJuhctioning efficiently and' accurately. 
checks may take two forms: ^ 

•^ ^ * Reverification of quality control procedtifes " .*. 

• Totally new checks at various system points using new data. 

ReverifiCation assesses the performance of a systett) step that was previously 
measured by the contractor using different data or di^erent tcsst cases. The data, 
used by the OSFA monitor may be iiid^ent^iy collected (or gefi«^tec0 or 
"represent a different sample of actual cases than that seJecte4 by tiie contractor. 
•Independently collected (or generated) data are mwe co!((l:y^ uslr^ a hew sampte, 
of the data maintained by the cwitractor. Therefore, they should be used 
judiciously. " * - ^ ■ 

New independent checks represent different measures of system performance, 
than tho^ used by tl^ cqntractfi«- In the quality coo'trot plai^ As is the me when 
reverlfyihg procecfeir«, new tewdepende^ checks may also be made With either 
independently coUected (or generated) data or a subset of the contractor's data 
base. ^Kfepa^Mt <^J3iity, asairance chec^es^^ c^ l^fi^med oni 




• ,A rei^larly schejJuIecf basis \, ' 

• ^ ad hoc basis at the discreti(?n ot the roonitoni 

^This d^ciision wUJ vary bas^d on the type and cosi of the check. 

Since a major function of the quality assurance process is to monitor the 
quality fcontrW process, these two systems should be parallel in design and, to the 
«ctent >s*ible, developed simuitaneously. ■rherefo™. OSFA should, carefully 
evaluate the contractor's proposed quality control plan. In developing its parallel 
quality assurance plan, OSFA should specify the following? ' 

• items on the quality control reports that slK>uld be reviewed as part of 
the quality assurance plan * f 

f The role that -an ^-site mohitor and/or site vists will play in quality 
assurance |>rocedures 

• Parts of the quality control procedures that should be independently 
checked by ^^^^ - 

• Types of instruments to be used for independent quality assurance checks 
• and for evaluating quality c^trol procedures* 



3.0 SPECffiqQUAUTY ASSURANCE PROCEDURE / 

In a companion report to tiiis document. Advanced Technology, Inc. described 
the critical guaiity control procedures that^ should be required olT a contractor 
processing student -aid applications* In this section, a series (^ specific quality 
assurance procedures are detailed that parallel the contractor's equality control 
requirements. These procedures, as were those in the earJier quality control report, 
are )>rok;en down into^ the four major components of the processing system and two 
major functions that interface with these four components. The major processing 
components ar^: " \ , ' 

.Production control .; .•■ ' j '■, 

• ' 'Fiscal con tijol ■ . I 

^ • Software quality as^fance " . 
. / " -- m ■ ■■ Productivity' control. ■ ■ t ' 

Where appropriate, tl^cse Components are b^k^ down into subcomponents. The two 
.component int^f aces are: ' . V-,/ - ' J, v-.^ 

• Reporting process 

• Corrective action pr«»s^ * ' 

^ &t order to meet the r»eds of OSF A monitors for a detailed^.yet.cbncisei 
resoiHxe on specific quality asi^irance roles, eadi component, sutKrdmpohent, and 
^component interface is discussed in a non-narrative, summary format.. This format 
divides each individual description into similar sections: 

\t • Definition of the profc*Ming procedure 

• Goal of the processing procedure 
,. ♦ Activities to be measured 

-# Quality control measures 

• Role of quality a^irance. 



t 



The latter section, the role of quality assurance, dlscusse$l the specific 
•functions that can be, carried put byOSFA monitors to oversee the cjUality control 
process and to ensure th^ quality of the processir^ system. The list of quality 
assurance roltfS is not necessarily exhaustive. • It is meant to guide the monitor; 
through his or * her duties by providing a variety of major quality assurance 
responsifaiiUties. The monitor san, and should be encouraged to, build on this Initial 
catalogue* of roles. This list if not meant to Umlt the monitor in the exeoition of his 
or h^ respQnsibilities. ' . , 



. 9 
» 



Further, not all items oh the list of quality a^urance roles will be exequted in 
all processing contacts or during all phases of particular prcjce^ing contracts. The 
rnohitor and the appropriate si4)ervisor shoulS work together to sel^t the roles tl|j^ 
are appropriate to a particular situation. * , • 
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I PRODUCTION CONTROts 



Proc«i$s of inorUtoriilg the; f or^fficleKy and accuracy. 



1 \ '.V.' 



Coak 



Ensure that applications" iTidve from one pro^ductiori step to the next in th? 
least amount of time withptit sacrificing quality, and that products (such as 
Student Aid Reports tS^^ 3 and correction re<^ests in ?*eld are generated 
accurately. * < - - ^ T . ' ' 

Activities to be Kfeasured: 

information receipt (see Seittion ^.i.l) 
* Data entry (see Section 3.1.2) ' - / 

Dataedit (see Section 3.1.3) • 
Compute (^e Section 3.1.4) . - 
Document prcKiuction gmd mailing (see 
Corrections*procedures(stee Section 3.1.6) 



QuaUty C^trol Measures: ' ^ ^ 

Se* separate listing fpr eadi ^•Actlyity to^be Mcasuretf'V <m,ttw*folldwu^ 



Role of Quality Assurances 

See separate lislng for each "Activity to be Measured!* on the following pages. 



.1 iNFORttXtioN kkc^irrs 



^ The point wtere % initial ai^licatiopr collection, tdephone call, or data 
tape irom a MultipW Data Entry (MDE) site conies i^^^ 



Goal: 



Receive information and transform it into a form that be used in the 
: processing stream.- ■ ■ ' ':X\^'^i' ■■■■■ ' 



Activities to be Stasureds ^ . , 

, ' A'T'oynt of incoming information ^ 

Timejiness of preparing infor mati(^ for f urthifer processing 

Accuracy of sorting information fay type 

Accuracy" of cursory edit for completeness of application 



Quality Control Measures: 



Count of appUcaticms receiv^^^^ • 

Backlog of applications / 

Processing time from receipt to data entry - 

Count of appUcations flagged by cursory edit 

Count of the number of times a particular edit is used 

Percent of errors'ln sorting applications by type 



Role M Quality A^urance: ; . . 

• . Take an in^pendent sampfe fvcm each groi^iH^ of incoming <k«:utnents 

Cappllcatfonsr correspondence, cot;f ectKjn^ e^), assign an identif ykig 
number, and record the date and time of entry into- the system* Each 
document can then be monitored throughout the processing stream* At 
each produttiort step, the date and time of processing should be 
■ ^ recorded. This will aliow assessment of the timeliness and accuracy of 
the production process* , '. ' ^ 

" ■:■ r . ; ; 

♦ -Compare results of the <piality assuranc«j sample to t^^ 

sample. If major disparities exist, have gje contractor's qualify control 
manager justify the contractor's results In a written report. 

f Monitor the contractor's receipt .control sampling pcoce^. Be sure ^at 
an independent and objective quality amtrol team is performing the 
sampling. * ' / 



3.U Ir^bimAllON REC^^ (CantintittOs . <w 

• Compare €0unts af Incoming wU<^^^^ 

. steps. This can assess whether a signficant propbrtion of appUcajtionS 
are being lost during processing. • 

;^ • Obtain from the contractor informatil)n on th6 backlog of, applications, 
' the accuracy of any intdrmation Sorts, the number of applications 
flagged.^' "ttie cursory edits, and the number of times a particular edit is 



• •■1 



1 
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Process by wludi new data, usually from paper forms, are manually key 
entered into the comiHitd- system. Information from MD£ sites wUl be 
submittedonmagn^ctepe, rather tfia^ 



Goals.'- , : ■ 



^ Accurately enter data items into the processing stream so that subsequent; 
processing steps utilize correct m^^^ ' . - \: 



Activities to be Measureds > 

Amoiiht oi data being entered into the system 
Accuracy Of data entry 
Timeliness of data entry 



■ f 



<^iaUty Control Measures: 

• Count of applications ehix^dd into the system coiftpared with counts at 
various automatic pr<N»ising ste|*s ; 
Count and percentage 6f ^rnirs by keystr<rf<e, data element, and entire form 
backlog of appUcatibns i . 

Processing time from acceplpce of c^^ by keyp(«idt to conipletion of data 

1 :. . : ... ■ • ■ • 

|. .1^ . - . ■ - - ■ ..... • . . . 

: ■ ■ ■ ' . V ■ , J . /. ■ ; ^ . ■ ■ ^ ' 

Role of Quality Assuramiet 

• Take m imfepmdent san^ie of various data input doaiments and 
compare trafw^ohned cteU to original documents for;^±iM^. Since a 
large number of forms and data itfents are involved, this is pbtentially an 
error-pftene area. Also, because this is the first step in using specific 
data needdd to run the processing system, accuracy is particuiaply 
import^t. If data entry is inaccurate, subsequent prqiiUctlbn steps, will 
be inaccurate. v 

. - ♦ Measure timeliness of ckta cntf y. by momtoring j6ie processing, $ime f rom_ 

^ data arrival at keypunch to transmission of <^ata to data edit staff. 



Compare results of the quality- assurance sample to the quality control 
sample. If major disparities exist, have the contractor^ qtKiIity controi 
marker Justify the contractor's results in a wrift^report. 



I 
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3.1^ DATA E^^^RY (Cpntim^i^ 



• f Monitor the ct>ntFactof *s <iata entry quality control sampling process^ Be 

sure that an independent and objecti^k'e quality control team is perlorm- 
. . ■ ■ ■ ing -the sampling. /" • '/ ' ,. ' ,t /' ■ ' ' ' "■, ' * . • . ' . ' 

• Compare the counts of applications entered into the system with cbunt!5 
►I at various automatic processing steps taken by the contractor. This can 

^ess whether "a significant proportion of applications, are i^if^ lost 
'toin§ or after data entt^^ ^ 

• Obtain from the cdntr.ac:tpr a count of $he number of ^piicaitionis 
' backloggcd at the data entry point. Th^ backlog is a measure^^of the 
, efficiency and timeliness of ^^ d^ entry. Efficiency and timeKness are 

patticularly important b^csi^se data «ntnr is a potential preceding 
Bottleneck since most ^ai^ are entered nli^nually. Backlogs should be 
measured durii^ various intervals in ^e processing cycl^ 



XtJk^ DATA EDtTj 



Process by whidi data trpm forms altered; intone system are checked for 
accuracy ami consistentcy. Edits ocoir at various points in the production 
prbce^ afid may be manual <»- autoroa « 

Goal-., ■ . ;, - . . ■ . '» ::' 

1^ subsequently correcti inaccura^ Of Inconsistent data to 

• computing.a 'Student Aid Ihdex CSAl). 

. , .' , ■ ' ^ ^ , 

Aenjyittes to be 

Frequency of errors identified by edits 

.Accuracy oledits, ,.' , , ' ■' ■ /■ ,, , . ''r ■ 

Accuratey of responses sent to applicants resulting from an eiH-or identified by 

* the edit procedure • « 

Timeliness'^of the edit/respon^ process 



Quatit9^CoiitrQi;Mcdsui^ .:.„■"-:'-.": 

Count of the number of times an edit identifier error 

Count 0f transactions per applicant . - ' . 

Cbwit and percent of applications with errors identified by edit procedures 

Percent of e<iit4dcntified errors tfiat are actual errors , . 

Percent of error responses that are appropriate to the type of errors detected 

Percent of applicants submitting corrections 

Percent of appBcante ftdjmittlns vall^iation of reacted da'te 



JtcOe of Quality AsSKirsnc^s . . 

w ■ . , ■ ■ . 

• Rurt an independent sample of documents throyjgh the edit procedures 
and compare results \jfith those of a separate manual edit* this will test 
whether only actual errors are being flagged> Alternatively! develop 
several test cases, some with errors and K>me without. Assess treatment 
of test cases by edit procedures. 



A. 



Take ah ^dependent sample of responses sent to applicants as a result of 
errors idwitified by the edit procedures^^' Be sure responses accurately 
match the type of error. 



V 



• Compare results of ^ c^iity assiarance sample to the quality control 
sample. If major disjSarities exist, have the contractor's quality control 
manager justify ^e contractor's results in a written r^ort. 



3a^ mTA BDrr (ContifMied9t ^ 

• "Monitor the contractor's data edit quality control sampling process Be 
sure that an indepiHident and objective quality control team is pe^orm- 
ing the sampling. 

• Obtain from the contractor county of the number of times an eiflt 
Identifies an error type of edit. Edit procedures rejecting an 

. unusually high percentage of cases may require correct 

. • C»»tain from *the contractor counts of transactions per apjplicant. This 
indicates the potential of edits to reject acc^tablfe data. 

' " ' . '., ■ ' • ' . - ■ " • ' • ■ - ■ 

• Obtain from ^e contractor Ae percentage of applicatibas/?ith errors. 
A percentage of error higher than that established may suggest the need 
for corrective action in application dcsigm 

' • Using documents assigned an identifying number at the information 
receipt stage/ measure turnaround time between edit and response to, 
: edit. If only a few such documents are flagged by edits, take a Separate 
sample fr^m documents awaiting edit. 

• pbtain from the contractor ^ the percentage of applicants submitting 
cofr^tipns and the percentage submittihg vaUdati 



3.1^ COMPUTES ; « 

^ Automated process which input <iata ar^ transformed, and the r,equired 
output data dor esianiple/ a Student Aid fiKlest tSAll in Pell or the funding 
level in CampuS'BasedS i$ calculated. 

■ ". Goals ■ ■ ^ ■ V ■■■ 

: ■ ^ ■ " .:■ ^-^ , ■ ' ■.'■„■• ■ ■ , ' , ■ , ' 

Accurate dbmputation 61 output da^. 

^^ ;Activlti<»t)p\be,Meaaired: ./^ •' ■■'.•■ /';'' ■ , 

V Accuracy^ of ^the Compute Step 



Quality Contn4 M ^ 
^ Peixeht of output da|k^w 

^ ■" • .: ■„ ■■. • , ^ • ■ '', - .. ' ■ . .'^ 

'tt(>leoiQiiaiIty;Ais»iranc» 1^/'' ■ '[-'^'■^^r-"s \- 

.... • ■ • • . ■' : . ■ , ■ . .t ■ 

♦ Take an independent sample ol applications and-infiantslly repUc^^ 
putation of output* C^pare results of compute 

catlofi. AecUr^ of computes shmild he measured for ^varloys volumes 
of applications. 

• Gcffppare results-of the ' quaHty askance ^tmp^ to the quality control 
sample. K major disparities exist, hav* the conti^ctor*s qiality control 
manager justify the contractor's 'results in a written report* .< 

^ • Monitor the contractor^ a>mpute step quality om^l sampling 

Be sure that an independent and objective quality control team is 
performing die sampling. . 
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3.1^ tKXrUMENT PRODUCTION AND MAIUNGs ' 



ArcK^ss 6f p/oducing ah oatput docun^t such as a Stud^t Aid Report (SAR)> 
' corrections request*' iett^rt or other document aiKixinaili^g it to the correct 

'^'a|)iiUcaiit.'; , ■ ; 



Produce a docum^t 
document and appropria' 




accuraw Information and mail SAR or other 
tir^ materials to the correct applicant. 



Actiivfties to be Measured: 



Number of, output docum^ts produced 

Nun^er of oiftput documents niailed 

Accuracy of output documente,' 

Timeliness of output document production and mailing 

legibility of all output documents 



QiialHy'Controritosarestv^^^ ^ 

Count of documents produced compared to count of documents mailed 
Ccwnt of documents mailed compared to count of applications ' at other 
production points ^ / 
, Percent of documents wi^ ir^ccurate of missing information 

Percent of documents mailed with incorrect accompanying materials 
Ptocessing time from compute procedure to document mailing . » / 

Percent oldoamientsiMegib|e / 



ERIC 



Role of Qualitsf As»iratK£S ' ' 

• Take an independent sample of specific output docum^ts right before 
maiUng. Check to see tiiat information on the documentfis complete and* 
legibler and that the appropriate supplemental material accompanies it* 
Repeat process for all categorizes of documents produced. 

• Compare results of the quality assurance sample to the quality control 
• sample. If "major disparities exist, have the ^contractor's quality coiitrol 

ni^nager Justify the contrac|or*s rekilts in a written r^>ort* 

e " Monitor fee contractor's pr oifciction andytftalHr^ quality control' sampiirtg 
process.^ Be sure that an |m^>endent»and obje^ve quality control team 
is perfocmihg the samplii^. ' 
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3.1^ DOCUMENT PRODUCTION AND MAIUNG IContiiwe^: 



• Using documents assigned ^ identifying number at tfi® information ~ - 
receipt stage, measure total turnaround time from, information receijJt 
to maiiing. >Since mailing marks the end 6f the ppoduction control^ 
pFocess^this measures total production elficiency. * - ,/ 

• Obtain from the contractor counts of output docCii|ients pjrcKiuced and 
V dutput documents mailed by various categories (e.g., SARs, corrections 

requests, letters). A comparison of these counts provides a control total 
for the whole system. ^These counts shoiild be compared tb those of 
other production points to ensure that nothirtg is^lng 



r 
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Process of dcc«ptir^ new data sent in r^ponse to a correcttons requesff 
ujxiating imfividual' rc^orcb^^^^^ r^ ihforfi^tipnt 9nA^ prckfeicing new 

outpm docu<hents« ^ si^gestedf coirectic^ ocxur atrv^ious points in the 
processini^ syste^^^ v 



GoaU 



Whfen a correction request is sent out^ process the correction in an efficient' 
..nianner. , ,' , ■ \ 



Activities to be Measured: 



Amount of^rrections received 
q Timelm^s of reentering corrected data into the system 
. Accuracy of processir^ corrected cteta 



(Quality Control Measures: 

Count of corrections receh^ed - 

.Count oi types of correctioos received 

Backiog of corrections * ~ 

Processing time from corrections receipt to data entry 

Number of times an applicant^ record must be corrected 

Percent of error reentering corrected data 



Role of Quality Assurances 

■V • ■ . ' ' ■ . ' * , 

• The indepenctent ^mplir^ process used during the kilormatlor^ recelp 

^ sta^e should be repeated at the corrections stage. This'' wiii ailow^ 
mtmitoring of corrected data throughout the processing stream and 

. assessment of the timeliness and accuracy of processing corrected data. 
Accuracy of entering corrected data will be ijieasured by a sample oi 
;documents obtained at th^ data entry stage. 

• Compare result^ of the quality asmiranc» sample to the quality control 
" sample* li m^^ 

. manager justify t» contractor's results in a written report. 

^« - Rfonltof the contractor^ correCtJom <|uality^ control sampling process- 
Be sure^TPtat- an. indepe^eat and obfective c^iafit con^t.team is 
performing the sampling.* 
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34.6 CORRECiniONS (OmtimiecOi 



Information on counts by the types of corrections received should be 
obtained from the contractor. This wiU help identify the need for 
cofrrective action in sud\ areas as application formats .* 

.Information on counts of corrections Teceived, backlog of CGrrections, 
and number of t«nes an applicant's record must be corrected 'should be 
obtained frpm the contractor. . 
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FISCAL CONTROL*. 
Process of nuKutoring act^ processing costs agi^^ b(^g«ted a»sts. 



Gaat: 

Ejuscute processir^ procedures in a poisrtfeffective manner without sacrifick^ 
.y . quality. ^ ' ■ ' ■ [ ' - . • . • • ■ 

■, , , . , , ■ ■ ■ ■ ■ ' ■ ■. •■ ' ' ' . , 

■. ^ * ■ i '-S ■ ' ■ ■' *' ' ' ' . '' ' ; . ■ ■ ' ' ■ ■ ■ 

Activities to be Measured: 

■ , ' ' ' ' ■ ■ ^ " ■ ' . ' ■■; , ' • ' " ' 

Total processing costs 

Processing costsTiy major cost center ' 

sts per unit of volume «• ' . , 



Quaiitjr Control Measures: 

Actual versus budgeted costf^by period of performance,- 
Actual versus budgeted costs by cost center - ^ 
Actual unit costs versus bud|eted unit costs 



Rote of Quality AssuraRoei 

• Monitoring roie varies depending m the type of contract award* For a 
cost plus fixed fee contract, ongoing fiscal monitoring is mandatory to 
avoid large cost ovef runs. ' For fixed co&| contracts» the contractor 
tyi^cally will not submit actual costs to ^e monitor on an ot^oing basis. 

t For a cost plus fixed fee contract, obtain from thi contractor actual 
costs t>f performance date and compare them to bui%eted costfc.r 
Significant cost overruns should be fkgg^ immedla^^. 

• Cost trends across {i^rlods ^f performance ^ould be analyzed carefully. 
Silice the volume of processing varies over the course of a y*ar, these 
figures wilt not t>e stable^ Costs for a particular perit^ of performance 
that exceed the budget should be flagged imnfediateiy. ^ - 

5 __Cpsts for .each perfprmanpe,|>eriod^^.s^^ be calculated by cost 

center. It is possible that a cost overrun may be isolated to one cost 
center. This type of ana^sis will ali6w corrective action. in this cost 
center. ' ■ ^ 




3JI FISCAL a>NTROL CCwitin^^ 

\ « • UWt costs must be "obtained from the contractor. This is important 
. beta^se volume varies over the year and processing costs vary with 
volume. ' 

< ■ - ' ., r 

• Costs should be categorized as fixed w vdriabl6 t6 help determine the 
level of costs by level of volume. 



1 ■ 
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33 SOFtWARE QUAUTY ASSiJtttAHCEs ' * V - ^ 

, Process of ending, withki r easofiable Jimhst tiiat aU software |$ accyrate 
■ Vand .efficient*. ■ ^ ■ --'''^f" 



Goal: 

' ' '\, , . ^■ ' V, ■. ■- : ■ ' : ' ■ 

Continual monitoring of software so that unnecessary delays and expenses 
resulting from discover ing software problems alter considerable' production 
has been completed are avoided. 



Activities to be Measureds - 

Accuracy of software on a day to day basis 

Reliability of software on a day to day basis 

Impact of minor software modifications on system performance 



Quality Control Meastic^ ^ 

Accuracy o| software components as determined by test data, validation, and 

verification tecim^c^es ^ 
Accuracy of software si^y^tems (compof»nts working together to produce a 

result) as determiiwd by test data, validation, and verification 

techniques., . ^' . ... 

Accuracy of system software (subsystems working together to produce a 

result) as determined by test data, validation, ana verification 
^ technijctues ' 
Cjaunt of the mjor^er of K^ftw^e failures per pr oces^ period - > . 
Backlog of applications due to software failures ^ 



Rote of Qu^ty Assurance: 

• ' Exercise direct oversight on a variety of software qtality control 

activities. Software qimlity assurance is particularly important because 
minor software errors can cause major problems in the normal processing 

* cycle, for example, minor sbft ware errors c^ cause output documents 
-«uch as SARs to be 4istribm^ 

f incorrect awards. In addition, an error detected in the production stage 
usually requires temporary ^utdown of the system,, possibly during paak 
operation. When the system is |n production all new appiications must 
be repn^es^d, a|ong with ^e backlog ^t accupsilated^ while Jhe 
system was shut down for correction, adrfihg to production costs. 
' Software corrections made quickly often cannot be tested as thoroughly 
as ^y should ber This condition ofteft leads to further jpompiJtatiohal 
problems. Therefpre, quality a^uran<:e mohitw* should be involved In 
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33 SOinrWARE QUALrrr A^imANCE (Cofitifuie^ 

• assessing the adequacy and felia^»ility of software during both 
development and operational stages. A recommended .division of 
responsibility between OSFA and the contractocu|or sq^tware quality 
assurance is shown in Appendix A. ? s 



Verify that contractor developed software meets OSf A specifications 
during software, design or modifications. 

- , • Oversee product evalimti<m, during which both the program code and 

system documentation are examined at the ^software impierfiehf&tion 
■ • 'stage.. . - ' " . . ■ 

* * Oversee the* development by the contractor of test data bases during 

software Implementation. Ensure that these fiata are of hi^ qu^ty, 
'^ and not merely of large volume, and test as many conditions and ranges 
' / as possible. A large variety of both invalid and valid data should fcte used. 
* Contractors often go to cjrtremes to create repetitious inyalid^ati while 

' , neglecting vaUdy^ata which tests air "Oie complexities of valid processing. 

• Monitor various contractor performed internal control techniques on 
. system performance during the software implementation stage such as 
f- record coiuits, batch totals, control totals, and limit checks. The 

Wternefl counting of records and the review of record counti! Is e 
" to keeping track of data within a system and verifyfing that the programs 
are passing, sortmg, or ufKlatir^ files properly. Counts are norrhally 
performed at least twfcce within a program— once when the record first 
. enters the program and. agabi at an appreciate point, usually 

^^^^ ^ belore j|ds pass<^ to another ^e or prografn. When a 

specified number of rSords are processed together as a batdi, as they 
/* are In Pell Crant processing, it is necessary to keep .tr^tck of these 

^* batches as tiiey flow through the system. This is done by calculating a 

batch total on a cfcita field common to all r^orcte &v the batch- "Rie 
batdi total, liatdi coynt, and batch number are then reconcited at 
appropriate points in system processing. Any discrepancies inust be 
I researched- ajfui corrected. Control totals are statistics cotr^ted on 
significant data el^v«snts before ^y are proc^^d by the systenu 
Totals on the same data elements ar« calculated again during, orv at the 
completion of, processing; by the software system. Control totaB give an 
estimate of the magnitude of the software errors In the system. Record 
counts provide similar inforiftation but are concerned only with the 
volume of errors? control totaU are concerned with the value of error&i 
• Limit checks in prograrijs can be used to identify c^ta and combinations 

; of data like)^ to be in^^d. The programs u» a series of s^^ 

Instructions that con^pare certain conditions and r^^ 
contains high and low limits for parameters. When errors are detected, 
special reports are created that report the error conditions to the 
C<ktractor. AppfOfriate oorrectlv« action must 
t>rf^- . ' ^ ' probktms. ' . ■ 
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33 SOFTW^QUAUTY ASSURANCE (ContinuecOs , 

• Oversee fornf\al saftware testing procedures , including subsystem 
integration testing, prototype testing, sy^^ 

testing. This role ensures that no serious errors go undiscovered* 
Subsystem integration testing checks the way system modules" are 
connected and assures that a subsystem can receive and pass data 
: V between individual modules. Prototype testing is used to check the 
correctness of the basic design .of program modules or siflj^stems. It 
can often give early insight into design problems which can then be 
■ remedied, making for less -rework in the final system. System testing is 
performed on detailed outputs All initial input and final output must be 
correlated to ail intermediate input and output- The system test should 
. confirm that the system specifications have been followed precisely and 
that the system is cdmpietely operational, lln^t, the system test should 
identify the program units or subsystem w^ere errors have occurred. 
Acceptance testing is similar- to ^Stem testing but it determines 
whether &r not the system dan be accepted and put into production. 
^ach sub^stem is evaluated separately and parts of the system tnay be 
accepted while others require further modification or corrections. 

1, , ,. . , . . ' ' . ■ , ■ ■ ■ ■ « ' 

' • Obtain from the contractor a count of the number of software failures 
and their impact iit terms of b^dUog^ of applications. ; . - 

; ' • Use checklist for software quality assurance (Appendix B) to monitor 
contractor's quaU g 

• Re<^ire Profit Manager to piersonally sign off on any software modifi- 
cations using the System/Software Modification Signoff Sheet <see 
Appendix CK This will ensure that OSFA is awarfc of all impejjdir^ maior 

'..software moclifications.' ■ ■ 2''- ■ ■ 
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%A PRODUCTIVITY COHTROU ^ . 

Process of el£icientiy carrying out all manual ta$K§ in ti» processlifis syrtcm. 



Gcials 



Maintain an etf^|ient balance between automated §nd manual functions so 
that the oyeraU pfoddfbtjon process is productive. , . „ : 



.Activity to be Measured: 

Telephone service Sectiort 3.^. U 
Correspondence service (see Section 3.^f.2) ' 
, Information receipt (see Section 3.1,1) - 
Data entry (see %ction 3.1.2) 
Document production and mailinp (see Section 3.1^5) 
Corrections receipt (see Section 3.1.6) 



Quality Control liteasuresi \^ 

See separate listing of each ''^tivity to be ^teasured^ , : 

lU)^ of 'Quality 'Assurance ' y 

• Monitor the overall productivity (output per unit of input) of each 
manual task* The contractor^ qtmllty control plan will also moni-tor 
|>rockjctivity of each staff person to assure that overall pr*iu#iEivity is 
' maximized. OSFA is hot interested in ind|yidi«d staff productivity. 

. • OSFA -shoul^ sign off oh proAictMty standards In the- contractor's 
quality corrtrdi plaa and assure tet.standards are betig met. - / 
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3.4a . tEl£PHONE s^vk:^ 



ProC&ss Qf fesp9nding to tiie questions, of pn^pective appiican'^ phoiKKi in to 



Respond efficiently to inqMlfles abowt the application process and the states 
: , of^in^vidual appUcatiohs* ^ ; 



: Actiyities'-to be Meaatreds ' ■':■■'■.):'■■■ ,^ 

• ■ ^ 
Amouat of telephone iriijuiries * • 

Timeliness of answering applicant questions 

' Accuracy of ansiwring applicant questions 



(Quality Control Measures:^ * 

(gount of telephone calls received 
I^rcent^^ of incoming calls placed oh hold V 
' Average leng^ of time calls are placed on hold ► 
Count of the number of calls per application 

Accuracy of answering applicant questions determined by monitoring of 
\',-:'.- telephone: Operators; :,.'„'';.;.:''■■ V:/; ■ '''-^ 



Rol^ Quality Assurances ^ 

♦ Independently monitor operators on' the telephone service to assess 
accuracy. Concern is with ov«^il accurgK:y and courtesy of the service, 
not the accuracy Sffi4 courtesy of ir^vldiai operators. The latter is a 
quality control concern for ^e contractor only. 

• Prepare a list of typical questions asked by Callersi as well as appro- 
priate responses. Have OSFA staff phone in these questions to the 
telephone bank and record the accuracy of responses, courtesy of the 
operator, length of time the call was placed on hold, and length of the 

„ entiri? phone cqayersatipij. Cmcern ^ wte 

courtesy of the »rvice, not accuracy arid courtesy of ihdividual 
operators. Therefore, results must be a^regated to the overall service* 
level. 



• Compare results of the* c^aafity assurance monitorihg to the quality 
control monitoring. If major disparities exist, have the contractor's 
quality control manager justify the contractor's results in a written 
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• Monitor the cohtractor*s telephone service quality control process. Be 
sure that an Independent and objective quality control team Is 
performing the quality control monitoring. > 

• CH>tain from the contractor a count of the nunrtber of. telephone calls 
received, the percentage of calls placed on hold, the average number df 
calls placed on hold, and thie number of calls received per application. 
The latter statistic will help indicate how well the overall processing 

; system works and how understandable procedures are to the appUcant. 
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,•, ■■■■ , . y , / ir\ 

3.*-2 CORR)5$PONDENC^ SERVICE: r 

Proqsss of responding to the questions of prospective applicant mailed to a 
corre^>ondence'bank» . \ " '^'V'.:: 



Respond efficiently to intjuiries about the application process and the status >^ 
of individual applications. ^ 



Activities to bie l^bastireds 



Amount of letter inquiries 

Timeliness ol answering applicant questions 

Accuracy of answering applicant questions 



Quality Control ISeasuress 



Count^of'ietters receivedv','' ' , ■ ,■' ' r'^V ■ '■■ / '■'^^ ^ 

Count of responses to letters ■ ..' ' ' '.^ ' \ : '. 

Average /rtumber of days before a response is njaiied ; ^ 

Accur^0^ of answering applicant questions cfeter mined by writing s^pie 

; iettics and monitoring Responses / ' ' ^ . 



1^- 



Role of <^iial£ty As^iraiices 



• Write sample letters that 'represent the diverse types of questions 
applicants may have about application procedures and the status of their 

• Monitor responses to sample letters for accuracy and timeliness of 
responses* * 

■ ' ■ ' 

• Compare results of the quality assurance monitoring to the quality 
control monitoring. If major disparities exist, have the contractor's 
quality control manager justify the contractor's results in a written 
report.' •' . ■ ," ._. „ _ _ ™ ' " ' _i 

• Monitor-*the Contractor's correspondence service quality control process. 
Be sure that an independent and objective quality control team ls\ 
pc*"fjOrmii^ quality contro 
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# Obtain from contractor a count of the number of letters received* ■ 
number of responses mailedf the average number of days before a 
response is mailed* - ^ 
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REPORTING PROCESS: 



pFop^ of Indicating to OSFA that measures of the qualily aind efficiency of . 
production iaU witi^ established stan^ 



, Conveys to OSFA pertinent information about the processing system in a 
timely and accurate manner without overburdening managers with usel^ss^ 
^statistics*.- ' ■■^',:-> 

Activities to he Measured: : /\' 

Responsiveness of reports • * ^ 

Timeliness of re^fcorts, :'\ - 

Clarity of information presented in reports ~ 



Quality Control Measures: 

■ Q 

Elapsed time between 

Qualitative assessment of whetiier reports sufficiently flag processing 

■.■problems \ ^^. ■'/;;„■.;:■'■■-'' ■ 

Role of Quality Assurance: 

Estahlish "tickler file*' Which lists ail reports and other deiivarables due 
from the contractor and their delivery dates to OSFA. v 

e Send follow^ correspondence each d&y to amtract«r if deliyerabie or 
report is past due according to "tiek|er file." This should help reduce 
time lags between pr<Kluction of data and reporting. ' 

#*' Analyze all reports as to whfether they are directed to the proper 
manager and con^ih an appropriate level of statistical aggregation. 
^ Managers should not have to wade through excessive Information to find 
the data they are intere^ed hi. Top level managers should be sent 6nly 
" "siimjihary re^ rejpdrts on special f^ocesslhg problems. Lower levil 
noanagers^ who monHor relatively smaU, well deiifWfd parts of .the 
processing system should receive more detailed information. 



3,5 

• Assess whether f«p^ trend data as well as cumulative 
: " statistics. Comparisons o! trends are an important source of identifying 

potential processing problem areas. Further, statistics should fnclude 
r both averages and fre€|uency distributions since the former may hidfe 
\ ^ends for outlying casies. 

• A^ess whether a clear **sy$tem alarm" capability is u^sed in reports. 
4, _ Clear visual cues ^ould be used to alert managers to system problems. 

method for highlighting data that are well, beyond established 
standards ^ould be provided. iSuc^data ^ould pe flagged using a visual 
device such as an asterisk or bold l^pe of should be pyt in an exceptions 
^ report. Alsa, graphics should be used showing linear boundaries for 

- „ expected behaviors. In addition to visual cues* explanatory material 
must be provided for all out^of-range data and data that are 
pr«gre«ively approaching minimum performance standard levels. 
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,3.6 a>RRECTlYE ACTIONS PROCESSs 



Fonnai prdcei&ircs to oiodify the processing system in^ order to eUmin^te 
program errbr^v ^' . 'j , ^ 



I , : Efficiently assess aiternat^ve program moidifications, select the preferred 
I .^itemativet and implement modificaiions in a timely mannert 



Activities toibe hUtasxstt^ 



Timelinfijss of implementing program modifications 



Quality. Cor^tf oi Measures: 



Elapsed time from identification of program error to implementation of 
■ systjsm modification , ^ 



lloie of Qi»|[ity AssuraiK . ^ ^ ^ * 

• Oveil'see corrective Mrtion^obess in order to assure that modifications 
are foade in a timely manner and In a ^ay acceptafole to OSFA. 
Prbposed' corrective actions should be reported to OSFA for review in 

. the system/software modification signoff sheet shown in Appendix C. 

• Establish* standards for d^cidMg^w^wn corrective actlpns should be 
initiated by the contractor and by OSFA, . ^ 

1^ iEsjabMsh, criteria for the ^pf oval pro<»ss^-^ IMder 
different situations, corrective action propotofe may be implemented 
without OSFA review, with review by tiie QSFA monitor only, or with 
additional OSFA review. 
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fJI . GENERAUONGOING QUALITY ASSURANCE PROmDURES - ♦ 

There are a series of quality assurance roles th^t are either |eneral in nature, 
ongoingi or interface with severa/'proces&ii^ functions. They are summarized in 
this separate section* 

Addttionat Quality Assurance Roi^s 

• Provide input into the design of the contractor's quality co/itrol plan- 
This will facilitate the developmertt of »a qtality assurance process that 

/ parallels the quality control process. " 

■ ■'• : , / . , ■ ,, ■ . , ' ; - : ■ _ 

• Independently est^lish performance standards fpr each processing 
function. These "standards should ^ challenging, yet realistic, and based 

V upon historical data oa^lrocessing efficiency* 

. • ■ Monitor the contractor^ qu&JIty control procedures to verify the 
processes used to produce quality cpntrol statistics and reports. 

• Observe the^ production process on a regular basis in order to identify 
potential problems and/or deficiencies. ConH>are system performance 

/based on this observation to information in the quality csmtrof reports; 
'. ■ ■ ■ ■ - » • ', . ■ , ' 

• V Identify mparts of ^e quality control process that should be independently 

^C^iecked by quaMtyassurance procedures. ^ 

■ ^ • ^ W^fftSfy types of instruments to be used for independent quality 
^- :^ assurance checks and for evaluating quality control procedures. 

• OMfiduct independent qimlity aswande checks on system performance. 
. Data from these checks should be cqmpared to quality control data. 

• Compare production measures from the quality control process and 
quality assurance checks to estab^Mied standards." If performance is 
outside of tolerable range, report this information promptly to the 
appropriate OSFA mamiger, % * 

• Set up table shells and appropriate graphics to report quality 
control/quality assurance findings to Director, Division of Quality 
Assurance. 



Prepare and submit test applj||atioh cases on major application forms. 
Responses to these test cases Cf or example, corrections requests and 
receipt of SARs in Pel0 ^ould be- monitored and the accuracy and 
timeBnesI 6f responses should be recorded. . , * 

.For test application cases returned for corrections, prepare revisions. 
Responses to these revisions should be monitored and the accuracy and 
timeline^ of responses should be recorcfasd* f . 



Set up network ol quality asstirance field vbiufiteers (appfsroximately 
iO-12 people) made up of students and staff from financial aid offices, 
the National Association of Student Financial Aid Administrators 
ihfhSFAPdp the Nationar Association of State Scholarship and Grant 
Programs. In ASSGl*), and the National Council^of Higher Education Loan 
Programs (NCHELP). Have vokinteefs send in test, application cases and 
monitor responses. \ 

Conduct random survey of applicants' responj^s to the quality of the' 
-processing system. A postcard survey could be designed and sent to 
applicants in various stages of processing. Clearance will be necessary 
prior to imjilementing the survey. - 
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5J) SUMMARY €HECKUST/TEAR-OUT SHEET OF QUAtlTY ASSURANCE 



PROCEDURES- 

* 

Activity 



'Input to desigii of contractus' QC plan 



EstabUsh independent peri^ormance stan- 
dards ■■'V , ■ i .■' 



List deiiver^ieSf due dates for each ^ 

Establish "tickler JJle" listing reports, 
deliverables, and due datds 



Send follow-up correspondence fo;* 
deliverables/reports past due according 
to "tickler file" 

.,;; v ; - 

Monitor contactor's QC*|>r^ 

. ' ' '■ ' ' * ^ , . , " ' . 
Rewffiw QG reports | 

Observe •pro<iuction process 

Compare observed system periormance 
to QC reports 



Compare measures from QC' reports to 
standards v 



Assess timeliness and responsiveness of 
reports 

Assess whetber rej^rts are directed to 
prdpeF^ manager and contain correct lev^i 
of statistical aggregatitwi , 



Responsible 
Staff 

Team mtfnbers 
Team menders 



Team leader , - 
Teitm membe^ 



Team members 



Team members 
Team members 
Team members 
Team members 

Team merpbers 

'■ > 

Team members 
Team rhemfeers 



Check Wh«i 
Coropieted 

■ . /',, — .■ 



5.0 SUMMAk^ CHEClCUST/tEAR-bUT §HEET (CofitiiwecD 



Activity 



Assess whether reports include clear 
visual cues and "system alarm" cafmbUity 



Identif:^ parts of QC system lor mdepen 
4ent QA checks - 



Res|K>nsibie 
Stiff 



Team members 



Team leader 



y 



Chedc When 
€k)mpleted 



Identify types of instruments for indepen- 
dent QA checks 



Team leader 



Reverif y QC procedSres 



Set up table shells for internal QC/QA 
reports to Director, DQA 



Team members 
Team leader 



Initiate new QA checks on system 
performance and compare, wi*^ QC data^ 



Team members 



Compare QC and QA data to standards; 
report problems to Director, DQA 



Team members 



Prep^/submit test appllca-fion forms 



Team'm^Fitos 



Prepare/submit test letters 



Prepare test questions for phone bank 

Phone in test questions? monttor 
responses 



Team members 



Team members 



Monitor regular operations of telephone 
bank 



Team members 



Tearh members 



Prepare/submit revisions fdr t'est 
applications returned for 
■corirecti<»ts reqi^st ■ , ' . '* 



Te«on members 



5.0 SUMMARY CHECKLIST/TEAR-OUT SHEET (CommiK<|> 

. Responsible : 
AcMvltv Stalf 



Set up QA field volunteer network 



Team leader 



Check Whai 
Completed 



Prepare/conduct applicant satisiaction 
survey 



Team members 



Sample imromihg documents; assign ID 
number; record time in 

Monitor sampled incoming documents 
through each production step 



Team members 

■ ■ , . ^, ' ' ■ ■ ■ ' - ■ 

Tezwh members 



Compare QA monitoring results to QC 
monitoring results at each production 
step 



Team members 



Monitor contractor's QC sampling process 
at each production step 



Team members 



Obtain from contractor 
counts, percentages, backlogs, 
^ measiffes of accuracy, and measures of 
timeliness at each production step 



Team members 



Take sample of documents at data entry 
st^e and measure accuracy of process 



Team members 



Run test c^ta through edit procedures 
and assess accuracy 



Team mwnbK-s 



Take Sample of responses sent to 
applicants as a result of edits 



Team members 



Take sample o| applications and manually 
replicate output data computation 



Team members 



Take sample of output dpcuffients tefore 
mailin§^ check If information is 
com{rfete and legil^ ^ ^ 



Team ho^bers 



5.0 SUMMARY CHECICUST/TEAR^OUT SHEET (CorttinUedO 



Activity ; Staff 



. , ' ■ ' , , . ^' 

Sample responses to corr ettion requests; 
assign ID number? record time in 

■\ ■ ■ . ■ ' ' ' ' ' ' ' ' > 

Monitor «mpied rei^onse? to correction 
requests through ea^ production step ' " 

For CPFF contracts, compare actual costs 
to budgeteid costs »- 

Use "pieckUst for Reviewing Software" 
to monitor software perf drmance 

Assure corrective actions are made^ in 
timely manner 

Establish standards for deciding when" . 
xrorrective actions should be initiated 
by ciftitractor and OSFA 

Establish criteria for approving 
corrective action requests 

When appropriate, request contractor's 
Systcsm/Software Modification Signoff 
.Shedt ■ 

Prepare intenial OSFA QA reports for 
Director, t>QA 



Che^^Fhen 
Cemifrleted 

Team member^ ■ . ' ■ , ' . ■ 

Team meml^ers ' ; ■. ^" ■ 



Team memfclers ■ . ■ 

Team membfl|*s . ^ ■ 

' ■ ■ . '■' „■ . "'. ' V-' 

Team members ■ ^ 

Team;.m«itibers ■ . , . \;. ' 

.■Team'.men^>ers\' . \- 'x-^- ^ 



** , * . 



Team leader 



APPEf«)DC A 



DIVISION OF RESPONSBEUmr BETWEEN- CONTRACTOR AND 
OSFA FOR SOFTWARE QU^UTY CONTROL/QUAUTY ASSURANCE 
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AcnviTr 



PfdMARY RESPONSIBILITY OFs 
GONTRACTOR OSFA 



Software Design Stage 
Design Verification 
Design Validation 



X 
X 



Software Implementatian Stage 

. Pro?hict Evalti^^ 
. Structural Prografnming 
Technology 
Use of Drivers 
Test Data Bases 

Internal Control TechiTkl$j«t > 
. . .Record; Counts . 
; ")':; Batch Totals , - 

Control Totals 
■..1^ ■.; LirnltChedc^^ 

Ha^ Totals 

Software Testing Stage 

. . . ^ 

Informal Resting 

^ Fdrmal Testing 
Subsystem testing 
Prototype Testing 
System Testir^ 

- Acc«Pt^«?j!?5t|l!g . 



X 
X 

x 



X 

x 




x 

X 

X 

X 



X 
X 
X 
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I APPENDDCB 

MONfrok»S CHEOOIST FOR REVmWING SbFTWARE 
DEVELOPED BY CONTRACTOR 



1 
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sbrrwARE reviev cheooist 



Code Examinaticxi 



Check When Requicem^5t 
U Satisfied 



Naming conventions used 
Modular structure 
Proper .indentation 
ifeil commented 

Appropriate edit and limit checks 

Internal controls used 
(record counts, batch fctals, 
control totals^ hash totals) 

Routines for error handling 

No inexecutable code 

No gfoss inefficiencies 

Meaningful data and procedure . 
names where no conventions fi^Jply 

togtcaidata definition 
organization , ' 



i 



•.L-i' 



S0FT1?A$«E REVmw CH^aOIST (COMTINUED) 



Docunksitatioii Revi^ 



. Check When Requirement 
_____ Is Satisfied 



■ ■. i'. ' ■ 



AU inputs documented 

All outputs docunnented 

AH rti^ programs docunjented 

Ail subroutines documented 

AU online screens^d displays docume^ 

All abnormal termination and resort 
procedures documented 

AH error messages documented 

AH batch jobs documented 

Ail \itllity software documented s 

AH proprietary software ctecumented 

AH required system libraries documentkl 

(Operations procedures documented 

Cross-reference provided ' 

Output disposition documented 

AH run-time parameters documented 

AH files documented 

AH record formats documented ^ 
AH dat^ elemen^documenJed • 
friterfa^es with other systems documented 
System werviews provided — — " 
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APPEI«3IXC 

SYSTEtt /SorrvARfi MOTIFICATION SICNOFF SHEET 
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SYSTEM/SOFTWARE MODIHCATION SIGNOFF SHEET 



3YSTEM/30FTWARE PRpBl-EMi 



Date Problem Wentifieds 



RECOMMENDED CORRECTIVE ACTION? 



Scheduled implementation Date; 

•0 



.1*,: 



SIGNATURE OF CONTRACTOR'S ^ 
W03ECT MANAGERS v' 



•Datt: 



SIGNATURE OF OSFA»s 
PR03ECT MONITOR: 



Dates 



-I 



s ■ 
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APPENDDC D 



OaFA INTERNAL REPORTING SUMMARY FORM 
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Monitor*sNamo 



Today's Date 



Reporting Periods From, 



KEY MEASURE 



OSFA INTERNAt REPORTING SUMMARY FORM 
SECTi™ 1 « I^EY MEASURES 
KEY-ACTIVITY #i KEY AcVlVlTY #2 KEY ACTIVITY #3 



KEY ACTIVITY i » 



Volume Iniormation 

..''!. , . ' . --^ , . " 



Timfe (Average Processing/ 
(Resfio^ Ti«»€) 



Cost Information ^ 
Basetim Estimate 

Actual Cost 

Error Rate . 
Stanctard 

Actual . 

Other Key Measures 



This Last Year This Ust Year This Ust X6ar This Last Year^ 
Periwi Period to Date Period Period to I3ate Period Period to Date Period Period to Date 



Standard 



this iE^eriod 



Xast Period 



Year to Oafe 



I 



^ROCESSB^G FUNCTION 

I. PRODUCTION COIVETROLt 
A> bifordfiation Receipt 



Q^A INTERNAL REPORTING SUMMARY FORM 
SECTION 2 - EXCEPTIONS 
MEASURE OUTSIDE- TOLERANCE ' 



STANDARD 



ACTUAL 



f 



Data Entry 



C. Data Edit 



o 

ERIC 



i 



58 



59 



OSFA INTERNAL RISPORTING SUMMARY^I^ 
SECTON 2 ^ EXCEPTK»IS (CONTINUED) 
ftiEASURfiOUTS©E « STANDARD 



AGTUAL 



U PRODUCTION CONTROL (CONTINUED): f 
■ D. Compute ■ ' " ■ — 



1 



I- 



F. Corrections 
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V: 



OSFA INTERNAL fmPOR^G SUMMARY FORM 
V ACTION 2 - EXgEPnONS CCONTH4UED) 



PRCXinESSiNG 
2^FlSqALCONTROt ' 



MEASURE OUTSIDE TOLERANCE 




T 



J. 



■w 




7 



X SOFTWARE QUALITY ASSURANGfe 



er|c 




OSFA INTERNAL REPORTING SUMMARY FORM 
SECTION 2 - EXCEPTIONS (CONTINUEE^ 



PROCESSING FUNCTION 



MEASURE OUTSIDE TOLERANCE 



STANDARD 



ACTUAL 



PRODUCTIVITY CONTROL 



Mahiiat Proce^iure A 



I 



Manual! Procedure C 



Manu^ Procedure D 
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SECTON 2 - EXCEPTIONS (COMTiNUED) 



PROpESSING FUNCTION 

REPORTING PROCESS 



MEASURE OUTSIDE TOLERANCE 



STANDARD 



ACTUAL 



7 



CORRECTIVE ACTION PROCESS 



o 
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- Monitor's Nanie 



Today's pate 



R^oirting Fertodt Fi-om 



ilk 



KEY MEASURE 



OSFA INTEIU4AL REPORTlfiG SUIIM PELL PRdGESSlNG 1 

SECTION KEY MEASURES 
kMV ACTlVlTYil KEY ACnVrry f 2 KEY ACTIVITY #3 KEY ACTIVITY f 4 

Applications ProcesslitR Corrections Processing Gorresoondence Service , ^ Telep|^one Service 
This Last Year 



:. Vdiume Information 

.put-' ^ 

jtt Time (Average Processing/ 
Response Time) 



This Last. Year This Last Ye^r This Ust Year 

Period Period to I>ate Period P^iod to JDate Period Period to Bate P^idd Period to Date 



-.1 




Cost biform&tion ; V 
Baseline Estimate - * 

> 

Actt^i Cost " 

;or Rate 
^Standard ' ; . 

ActuaJ l 

OttierTCey Measures — — 

Number of applications in system more than 20 days 
f_ NitfT^ of corrections in system more th^ lO.cfeys' . , 
.Percent of applicatiohs with more than 1 transaction ^ 
-Percent of edit identif ied errca-s that- are real errors ; . 



NA 



-Stalndard 



-This Period 



,Xast Period 



NA 
NA 



NA 
NA 
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